Conversation
| m, n = size(A); minmn = min(m, n) | ||
| k = trunc.howmany | ||
| 1 ≤ k ≤ minmn || | ||
| throw(ArgumentError("trunc.howmany=$k must satisfy 1 ≤ k ≤ min(size(A))=$minmn")) |
There was a problem hiding this comment.
Do we generically throw an error if truncrank is bigger than maximal size? Would it not be more useful if we consider truncrank as an upper bound?
There was a problem hiding this comment.
You are definitely right, I think I was still dealing with the fact that CUSOLVER doesn't accept that by default.
There is an additional question about if we should catch that and then not do any sketching at all, since in that case we might as well just do the compact decomposition, but I'm fine with leaving that for a future optimization.
This does raise the question if we should error for the case where the trunc.howmany is larger than the sketch.howmany.
The cuSOLVER interface does not have this problem, since they use k = trunc.howmany and p = sketch.howmany - trunc.howmany and require both to be positive, and it does seem like it is sensible to not allow people to specify an algorithm where the truncation rank is higher than the sketching rank?
This PR implements infrastructure for working with matrix sketches, in particular through Gaussian sampling.
The main additions are
left_sketch!andright_sketch!, which could be thought of asleft_orthwith a sketching truncation strategy (although not implemented that way).Additionally, this brings the
CUSOLVER_Randomizedalgorithm implementation on equal footing by using aDriverto select between theCUSOLVERimplementation and theNativeone.Sketching API
left_sketch[!](A; howmany, kwargs...) -> (Q, B)withQanm×kisometry andB = Q'·A.right_sketch[!](A; howmany, kwargs...) -> (B, Pᴴ)withPᴴak×nco-isometry andB = A·Pᴴ'.abstract type SketchingStrategy <: AbstractAlgorithm— supertype for sketching primitives.GaussianSketching(howmany; numiter=2, rng=default_rng())— the one concrete strategy currently shipped.Sketched SVD
SketchedAlgorithm{A,S,T,D}—@kwdefwrapper bundlingalg(small dense factorization),sketch(SketchingStrategy),trunc(TruncationStrategy), anddriver(Driver), to denote decomposing via sketch + decompose + truncate. MimicsTruncatedAlgorithm.svd_trunc[!]andsvd_trunc_no_error[!]now dispatch onSketchedAlgorithm.They allocate
(U, S, Vᴴ)sized for the sketch (k = sketch.howmany), check shapes, then callgesvdr!(driver, A, S, U, Vᴴ; sketch, alg, trunc).ε = sqrt((‖A‖+‖S‖)(‖A‖−‖S‖)), identical to the oldCUSOLVER_Randomizedpath.gesvdr!(::CUSOLVER, A, S, U, Vᴴ; sketch, trunc, alg=nothing)directly. It currently only acceptssketch::GaussianSketchingandtrunc::TruncationByOrder(i.e.truncrank) and ignores the inneralg. Requires dedicated overloads for the allocation of the output because of CUSOLVER conventions.CUSOLVER_Randomized(; k, p, niters)reimplemented asSketchedAlgorithm(; sketch=GaussianSketching(k+p; numiter=niters+1), truncrank(k); driver=CUSOLVER()). Non-breaking deprecationDesign choices/questions
1.
left_sketch!vsleft_orth!I had a hard time deciding between overloading
left_orth!directly or introducing a dedicatedleft_sketchfunction. For simplicity I kept it separate for now, mostly since this is easier to reason through while fleshing out the design, but there might be a reasonable point to be made to merge the two.2. Sketching as an algorithm vs Sketching as a truncation strategy
In the
svd_truncimplementation, I chose to have bothtruncandsketchas a keyword argument, rather than trying to fit this through the same keyword. Again, this is mostly for convenience while I was playing around with this, but we could conceive of a way to merge the two concepts, especially if we decideleft_orth!(A; trunc=gaussiansketch(...))is a reasonable approach.3. Allocating outputs
One of the things I struggled the most with is the issue of deciding what
initialize_output(svd_trunc!, A, ::SketchedAlgorithm)should return.The main issue is that we already are abusing this a bit for the
::TruncatedAlgorithmcodepath, where by convention theUSVhwe pass in is the one that will be used for thesvd_compact!inside, and is not equivalent to the one that is outputted.For
::SketchedAlgorithm, I therefore chose to pass in the sketched sizes, since that is some workspace that can be reused, and that has similar semantics, but it is definitely true that this is a bit unintuitive.However, the biggest struggle in changing this is that our automatic differentiation implementations hinge on the fact that everything passes through
svd_trunc!(A, out, alg)in the end, so simply not supporting passing in anoutargument has quite some implications for the remainder of the code.This is probably a design choice that we could consider revisiting for a v0.7 version of MatrixAlgebraKit, since it might just be that for these cases where the output size is hard to determine beforehand, it doesn't necessarily make too much sense to allow passing it in as inputs.
The alternative would be to start playing around with "reallocation" strategies, where we shrink/expand provided inputs if they don't have the correct sizes. This does however feel like it might just be a bit too convoluted for what it buys us.
4. Truncation error formula
This is the
‖A‖_F^2 − ‖S‖_F^2identity, written for numerical stability.However, I had to significantly lower the tolerances for any such tests, since the numerical accuracy of the sketching and the floating point errors are quite finicky for this.
This is of course the same discussion we had before, where it is not clear to me that
svd_truncreally should have the responsibility to provide truncation errors, andsvd_trunc_no_erroris a way more sensible default mode.Presumably this is fine as long as it is clearly documented?
5. Gauge fixing depends on the exact truncation path
In the current implementation, the gauge-fixing happens at the moment of decomposing the inner small core tensor, which will then still get unprojected onto a larger space through the sketch.
While it is somewhat straightforward to change this, it might be confusing that there are gaugefix keywords at multiple places, which don't have the same effect.
I'm not sure what the best way forwards is for this, but this might require some more discussion.
TODO